We humans exist in three-dimensional space. We know where we are within the 3D world that surrounds us. Turn your head to the left. Now turn it to the right. Look up. Look down. You have just determined your place within the 3D environment of the room.
Ever since computer graphics became interactive, people have been trying to create human-machine interfaces that provide the same sort of immersive spatial experience we get from looking around the real world. As one example, systems are known that provide users with inertially-sensed head-mounted displays that allow them to look around and interact with a so-called CAVE virtual environment. See e.g., Buxton et al, HMD's, Caves & Chameleon: A Human-Centric Analysis of Interaction in Virtual Space, Computer Graphics: The SIGGRAPH Quarterly, 32(4), 64-68 (1998). However, some kinds of head mounted displays can restrict the user's field of view or otherwise impair a person's ability to interact with the real world.
Flat screen televisions and other such displays common in many homes today can be used to display a virtual world. They provide a reasonably immersive environment with high resolution or high definition at low cost, and most consumers already own one. Some are even “3D” (e.g., through use of special glasses) to provide apparent image depth. The person watching or otherwise interacting with a television or other such display typically at least generally faces it and can see and interact with the virtual environment it presents. Other people in the room can also see and may be able to interact. These displays are wonderful for displaying immersive content but, unlike stereo or theater sound, they generally do not wrap around or envelop the user. Wrap-around projected or other displays are known (see e.g., Blanke et al, “Active Visualization in a Multidisplay Immersive Environment”, Eighth Eurographics Workshop on Virtual Environments (2002)), but may be too expensive for home use. Further innovations are possible.
Some non-limiting example implementations of technology herein provide a movable (e.g., handholdable) window or porthole display surface into a virtual space. Aspects of multi-dimensional spatiality of the moveable display surface relative to another e.g., stationary display are determined and used to generate images while at least some spatial aspects of the plural displays are correlated. As one non-limiting example, the moveable display can present a first person perspective “porthole” view into the virtual space, this porthole view changing based on aspects of the moveable display's spatiality in multi-dimensional space relative to a stationary display providing a contextual spatial reference for the user.
In one non-limiting aspect, a human-machine interface involves plural visual presentation surfaces at least some of which are movable. For example, a display can present an image of a virtual space, and another, moveable display can present an additional image of the same virtual space but from a potentially different viewpoint, viewing perspective, field of view, scale, image orientation, augmentation and/or other image characteristic or aspect.
The non-limiting movable display can selectively display an additional image of the virtual space and/or other user interface information (e.g., text, graphics or video relating or auxiliary to the images the other display presents) depending on the movable display's attitude relative to the other display.
In some example implementations, the movable display can act under some circumstances as a pointing device. When pointed/aimed at another display, the movable display's attitude can control or influence the position of a pointing symbol on the other display. When pointed/aimed away from the other display, the movable display can display a further image of the virtual space displayed on the other display, but from a different viewing direction that depends on the movable display's attitude.
Technology is used to determine aspects of the spatiality of at least one of the display devices in the physical world, and to use the determined spatiality aspects to present appropriately-viewpointed, -viewing-perspectived, -directioned and/or other characteristic images on the displays. As one non-limiting example, determined spatiality of the movable display relative to a stationary or other display can be used to provide relative spatiality of images the plural displays present.
One example non-limiting implementation provides an immersive spatial human-machine interface comprising at least one handheld display movable in free space; an arrangement configured to determine at least some aspects of the attitude of the movable handheld display; at least one additional display; and at least one graphical image generator operatively coupled to the handheld display and the additional display, the at least one graphical image generator generating images of a virtual space for display by each of the handheld display and the additional display, wherein the at least one graphical image generator generates images from different viewing directions images of different perspectives (viewing) from a same or similar virtual location, viewing point, neighborhood, vantage point, neighborhood, vicinity, region or the like, for display by the handheld display and the additional display at least in part in response to the determined attitude aspects to provide spatial correlation between the two images and thereby enhance the immersive spatiality of the human-machine interface.
Such a non-limiting example interface may further provide that when two images are practically similar, the image presented by the handheld display is substituted by other image(s) and/or associated information. In this context, “practically similar” may include or mean similar or the same or nearly the same from the perception of the user who is viewing the two displays.
In some implementations, the additional display comprises a relatively large stationary display, and the at least one graphical image generator does not practically alter the rendering perspective (or viewpoint), except perhaps some marginal, peripheral or border-situation look-around perspective shifts, for images generated for the stationary display based on the determined attitude aspects.